Step Away from the Pool

Lane Kenworthy University of Arizona [email protected] administer the “treatment” to some observations. Then we compare the degree of change in both (or all, if there are more than two) groups. With observational data we don’t administer the treatment, but in other respects the process is similar. Especially useful is that this design takes constant countryspecific differences (“country fixed effects”) out of play. There remains plenty to worry about: selection into the treatment and control groups, reverse causality, country differences in other things that change, getting the lag right, variation in effects across subperiods. But this is a good design when appropriate. When is it appropriate? When we have significant overtime change in the hypothesized cause, when there is variation across countries in that change, and when the change is mainly unidirectional rather than back-and-forth. Doesn’t this throw out a lot of interesting year-to-year variation? It does throw out that variation, but in many instances that variation has little or no bearing on our research question. Here’s an example. In recent years social scientists have grown increasingly interested in the effect of income inequality on socioeconomic outcomes such as health, crime, trust, and educational attainment (Burtless and Jencks 2003; Wilkinson and Pickett 2009). Inequality has risen significantly in some affluent nations. Figure 1 shows variation in income inequality change (vertical axis) and variation in income inequality levels (horizontal axis) for 20 countries. The range of values is the same on both axes. Quantitative macrocomparative analysis of the rich longstanding-democratic nations—“medium-N analysis”—is dominated by pooled time-series cross-section regression. I guesstimate that more than 90% of the medium-N papers I read in journals and as journal submissions use pooled regression.1 Quantitative data on many of the institutions, policies, and socioeconomic outcomes studied by comparativists first became available for more than a handful of countries in the 1970s. For a while quantitative macrocomparative research consisted mainly of cross-sectional analysis of single-pointin-time data or period averages. By the 1990s reasonably lengthy time series existed and analysts began examining overtime patterns. Pooling over time and across countries helped alleviate what had long been considered the achilles heel of cross-sectional comparative analysis: the small-N problem, which limits the number of control variables that can be included in a regression model. Pooled regression became the tool of choice. It has remained so for two decades. In my view, that’s unfortunate.